Using Amazon Mechanical Turk for Transcription of Non-Native Speech

This study investigates the use of Amazon Mechanical Turk for the transcription of nonnative speech. Multiple transcriptions were obtained from several distinct MTurk workers and were combined to produce merged transcriptions that had higher levels of agreement with a gold standard transcription than the individual transcriptions. Three different methods for merging transcriptions were compared across two types of responses (spontaneous and read-aloud). The results show that the merged MTurk transcriptions are as accurate as an individual expert transcriber for the readaloud responses, and are only slightly less accurate for the spontaneous responses.